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Abstract 

Under  Naval  Postgraduate  School  funding,  the  Principal  Investigators  have  conceptualized  and  are 
currently  validating  a  system  maturity  scale  called  System  Readiness  Level  (SRL).  It  combines  the  currently 
accepted  Technology  Readiness  Level  (TRL)  with  an  Integration  Readiness  Level  (IRL)  to  assess  a  whole 
system's  developmental  maturity  and  determine  current  and  future  readiness  during  the  defense 
acquisition  lifecycle.  As  a  function  of  the  TRL  of  the  components  and  the  IRL  of  the  integrations,  the  SRL 
scale  has  been  used  by  the  Pis  to  develop  system  development  optimization  models  that  can  maximize 
the  SRL  of  the  system  subject  to  resource  constraints  or  minimize  the  cost  of  development  subject  to 
attaining  a  pre-determined  SRL  value  within  a  time  constraint.  This  research  builds  upon  these 
foundations  to  create  a  systems  development  lifecycle  maturity  management  approach,  which  we  define 
as  Systems  Earned  Readiness  Management  (SERM).  We  envision  SERM  to  be  a  suite  of  management  tools 
which  can  be  used  to  manage  the  development  of  novel  high  technology  systems.  Current  research  to 
date  has  produced  a  scheduling,  monitoring  and  evaluation  tool.  In  contrast  with  Earned  Value 
Management  (EVM),  which  focuses  on  cost  and  schedule,  SERM  addresses  the  earned  readiness  or 
maturity  of  system  development  as  it  equates  to  making  strategic  and  programmatic  decisions  in  defense 
acquisition.  Future  efforts  will  be  directed  towards  complementing  the  scheduling  and  monitoring  tool 
with  additional  methodologies  such  as  Component  Importance  Analysis,  management  of  Key  Performance 
Parameters,  and  Interoperability  to  address  issues  associated  with  Multi-capability  systems. 
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1.  Introduction 

In  an  acquisition  lifecycle  there  are  many  factors  that  impact  the  decision  to:  develop  one  system  over 
another;  supersede  a  new,  more  functional  system  over  another;  determine  if  a  system  or  technology 
has  become  inadequate  due  to  changes  in  other  systems  or  technologies;  and  invest  in  the  development 
of  a  new  system  or  maintain  existing  systems.  To  examine  these  issues  in  engineering  design  and 
development  it  is  a  prescribed  practice  for  project  managers  and  systems  engineers  to  use  qualitative 
decision  methods  (Buede  1994).  However,  there  is  a  continuous  challenge  in  finding  methods  or 
approaches  that  produce  the  optimal  allocation  of  any  available  resource  to  minimize  development 
uncertainties  (Dillon  et  al.  2005). 

In  project  management  the  allocation  of  resources  is  frequently  done  with  the  purpose  of  creating 
individual  tasks  to  maintain  schedule  and  budget.  This  can  lead  to  a  focus  on  task-assignment  to  project 
scheduling  (Salewski  et  al.  1997)  even  though  the  ultimate  object  of  any  project  is  to  develop  a  product 
(or  system)  to  satisfy  a  customer.  Thus,  disconnects  often  emerge  between  the  priorities  of  the  project 
manager  and  the  systems  engineer  with  respect  to  the  optimization  of  the  design  during  the  acquisition 
lifecycle.  Furthermore,  additional  challenges  arise  from  the  allocation  of  resources  in  medium  to  large- 
scale  system  integration  efforts.  (Chang  et  al.  2001).  Salewiski  et  al.  (1997)  expressed  this  concern  for 
complex  interaction  to  minimize  cost  and  solve  a  time-resource-cost  tradeoff  problem,  but  the  focus 
was  still  on  tasks/activities  and  not  necessarily  optimization  of  systems'  developmental  maturity.  Dillon, 
et  al.  (2001;  2003)  developed  a  decision-support  framework  for  first  guiding  the  design,  not  the 
resources,  while  quantitatively  demonstrating  the  implications  that  constrained  resources  can  have  on 
critical  engineering  systems.  Although,  they  explained  that  much  of  their  work  was  focused  on  budget 
allocations  that  are  made  once  at  the  onset  of  a  project,  new  models  are  needed  that  allow  for  decision 
support  on  available  resources  throughout  the  lifecycle  at  key  milestones  (Dillon  et  al.  2005). 

Fundamental  to  these  challenges  are  that  in  project  management,  tasks  are  interdependent  and 
coordinated  in  parallel;  however,  engineers  cannot  afford  to  wait  for  complete  information,  and  are 
often  forced  to  continue  through  the  project  lifecycle  while  coordinating  design  activities  with 
preliminary,  ambiguous,  or  subjective  information  (Pich  et  al.  2002).  This  creates  a  tension  between  the 
project  manager  and  systems  engineer  (de  Haes  2006).  Unfortunately,  subjective  assessment 
techniques  are  human  intensive  and  error-prone.  Ideally,  assessments  should  be  based  on  system 
attributes  that  can  be  quantitatively  measured  using  system  metrics  (Yacoub  and  Ammar  2002). 
Therefore,  an  approach  that  can  combine  the  rigor  of  analytical  resource  allocation  to  the  subjective 
assessment  for  system  development  metrics  could  provide  potential  solutions  to  this  challenge. 

To  address  this,  the  Principal  Investigators  (Pis)  previously  described  a  Systems  Readiness  Level  (SRL) 
metric  (Sauser  et  al.  2006;  Sauser  et  al.  2008),  an  approach  that  incorporates  the  Technology  Readiness 
Level  (TRL)  used  by  the  Department  of  Defense  (DoD)  and  an  Integration  Readiness  Level  (IRL)  (Gove 
2007;  Sauser  et  al.  2010)  to  determine  the  maturity  of  a  system  and  its  status  within  the  acquisition 
lifecycle.  That  is,  if  every  technology  is  assessed  using  TRL  and  the  system  architecture  is  used  to  build 
an  integrated  representation  of  the  system  (e.g.  physical  architecture,  context  diagram)  in  which 
integrations  are  assessed  using  IRL,  a  metric  that  provides  an  assessment  of  a  systems  maturity  against 
the  acquisition  lifecycle  can  be  considered.  The  rationale  behind  the  SRL  developed  by  the  Pis  (Sauser  et 
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al.  2008)  is  that  in  the  acquisition  lifecycle,  one  would  be  interested  in  addressing  the  following 
considerations: 


1.  Quantifying  how  a  specific  technology  is  being  integrated  with  every  other  technology  to 
develop  the  system. 

Note  that  this  quantifier  should  be  a  function  of  both  the  maturity  of  the  different 
technologies  and  the  integrations  between  them  (as  dictated  by  the  system 
architecture).  That  is,  for  each  technology,  this  metric  should  be  a  function  of  both  TRLs 
and  IRLs.  Thus,  for  technology  i,  one  can  view  this  metric  (SRL),  as  "subsystem" 
measurement  of  this  technology's  integration  within  the  system.  In  a  mathematical 
representation:  SRL,= f(TRL,,IRLjj) 

2.  Based  on  such  a  metric  (SRL,),  SRL  should  provide  a  system  level  measurement  of  readiness. 

Note  that  this  new  metric  should  be  a  function  of  the  component  SRLs  of  each 
technology  or  in  mathematical  representation:  SRL  =f(SRLh  SRL2,...,  SRLn)  under  the 
assumption  that  the  system  contains  n  technologies. 

Thus,  not  only  is  SRL  a  quantitative  measure  of  maturity,  but  the  resulting  SRL  value  has  been  correlated 
to  qualitative  defense  acquisition  practices  (Sauser  et  al.  2008;  Sauser  et  al.  2008;  Sauser  et  al.  2008). 
From  previous  NPS-funded  research,  the  Pis  have  developed  models  focused  on  the  analysis  of  the  costs 
associated  with  the  drivers  of  the  SRL  so  an  SRL  Optimization  Model(s)  could  be  represented.  This  has 
resulted  in  two  models  that  have  completed  analytical  validation:  Model SRLmax  (Ramirez-Marquez  and 
Sauser  2008)  which  has  the  objective  to  maximize  the  SRL  under  constraints  associated  with  resources 
(cost  and  schedule);  and  Modei  SCODmir  (Magnaye  et  al.  2010)  which  has  the  objective  to  minimize 
development  cost  under  constraints  associated  with  schedule  and  the  required  SRL  value.  Both  can  be 
solved  using  evolutionary  optimization  techniques  such  as  the  Probabilistic  Solution  Discovery  Algorithm 
that  has  been  developed  by  one  of  the  Pi's  (Ramirez-Marquez  and  Rocco  2008).  The  mathematical 
representations  of  the  models  are  presented  below. 


Model  SRLmax 

Max  S/?L(TRL,IRL) 
s.t. 

Aj (TRL.IRL) <  /, 

Rn  (TRL.IRL)  <  r 


Model  SCODmin 

MinCW(TRL,  IRL  ) 
s.t. 

salt(trl,irl)>  a 

A  (TRL,  IRl  )<  rx 
«  (tKL,1RL)<  rn 


These  two  models  allow  for  decisions  to  be  made  regarding  the  current  and  future  developments  of  a 
system  within  the  acquisition  lifecycle.  Specifically,  both  models  identify  the  development  path  for  the 
system  such  that  the  pre-specified  strategy  (i.e.  maximize  SRL  or  minimize  the  System  Cost  of 
Development)  is  satisfied.  Thus,  the  solutions  show  which  technologies  and  integration  elements  to 
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mature  at  what  point  in  time  and  at  what  costs.  Given  these  global  optimal  solutions,  the  system 
development  manager  can  then  track  the  performance  of  the  system  during  its  acquisition  lifecycle. 

This  research  built  on  the  foundation  of  the  SRL  and  these  optimization  models  based  on  constrained 
resources  (as  depicted  in  Figure  1)  to  develop  a  decision  support  tool  that  can  enhance  managerial 
capabilities  and  create  an  acquisition  lifecycle  maturity  management  approach,  which  we  define  as 
Systems  Earned  Readiness  Management  (SERM).  Alternatively  to  Earned  Value  Management  (EVM), 
which  focuses  on  cost  and  schedule,  SERM  addresses  the  earned  readiness  or  maturity  of  systems 
development  as  it  equates  to  defense  acquisition,  so  variances  can  be  measured,  evaluated  and,  when 
necessary,  corrected,  with  the  intention  of  providing  answers  to  the  following  questions: 

•  What  is  the  resource  estimate  for  a  development  scheduled? 

•  What  development  has  been  accomplished? 

•  What  is  the  resource  estimate  for  the  completed  development? 

•  How  much  actual  resources  have  been  spent/consumed? 


Systems  Earned  Readiness  Management  Manageria|  Intervention 


Performance  Measures  / 
Variances 


SERM 


System  Development 
Schedules 


Optimal  Design  Solutions 


Figure  1 :  Current  and  Proposed  Research 

2.  Purpose  and  Focus  of  Research 

Early  indications  suggest  that  conceptually,  both  optimization  models  -  Model  SRLmax and  Model  SCODmjn 
-  can  provide  considerable  insights  into  quantifying  current  levels  of  accomplishments,  the  impact  of 
current  and  future  technology  choices  and  resource  allocation  on  the  development  process  as  well  as 
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their  implications  on  system  performance  during  acquisition.  Although,  we  still  need  to  understand  the 
validity  and  value  of  their  application.  Therefore,  the  first  research  question  is: 

Ql:  Can  the  SRL  scale  and  its  implementation  in  the  optimization  models  lead  to  a  more 
informed  allocation  of  resources  such  that  greater  system  readiness  levels  can  be 
achieved  at  the  lowest  possible  cost  and  earliest  feasible  time? 

With  the  optimization  models  validated,  some  arrangement  of  both  can  be  employed  against  existing 
system  development  plans  to  measure  the  degree  to  which  development  program  objectives  are  being 
met  at  certain  points  in  time.  Such  a  measurement  can  be  executed  using  the  proposed  System  Earned 
Readiness  Management  (SERM)  methodology.  As  a  management  tool,  SERM  can  also  be  used  to  adjust 
the  amount  of  effort  applied  to  the  development  of  each  of  the  components  of  the  system  so 
milestones  can  be  met  and  an  estimate  of  the  variance  in  developmental  maturity  can  be  calculated. 
While  EVM  evaluates  planned,  actual,  and  budgeted  cost  and  schedule,  SERM  will  evaluate  planned, 
actual,  and  budgeted  systems  development  levels  (i.e.  maturity)  and  use  the  allocation  of  resources  as  a 
means  to  make  developmental  planning  decisions.  To  establish  its  validity,  research  question  two  states: 

Q2:  Can  SERM  be  used  to  measure  the  development  status  of  a  system  and  calculate  the 
impact  of  alternative  budget  allocations  on  system  readiness  and  thus  lead  to  more 
efficient  distribution  of  resources  for  development? 

3.  Research  Approach 

The  development  path  for  SERM  proceeded  in  a  spiral  manner.  Each  spiral  served  as  a  foundation  for 
the  development  of  more  complex  concepts  while  also  transitioning  knowledge,  information,  and  tools 
to  a  practitioner  community.  Likewise,  each  spiral  was  also  closely  linked  to  the  previous  in  order  to 
further  develop  the  proposed  concepts.  For  example,  the  determination  of  the  validity  of  SRL  research 
becomes  the  basis  for  the  development  of  the  optimization  models.  In  turn,  examining  their  validity 
provides  insight  towards  the  development  of  SERM.  During  the  development,  verification  and  validation 
of  SERM,  it  is  anticipated  that  there  may  be  a  need  to  go  back  to  the  preceding  spirals  for  refinement; 
thus,  creating  feedback  loops  to  calibrate  the  concepts  and  tools  more  accurately.  Our  research 
questions  and  the  development  cycles  that  flow  from  them  are  reflected  in  the  approach  for  this 
research  that  will  consist  of  these  five  spirals:  Spiral  1  -  Validation  of  Optimization  Models,  Phase  2  - 
Development  of  a  Scheduling  and  Monitoring  Tool  (SERM),  Spiral  3  -  SERM  Verifications,  Spiral  4  -  SERM 
Validation,  and  Spiral  5  -  Methodologies  for  Multi-capability  Systems. 

3.1  Spiral  1  -  Validation  of  Optimization  Models 

As  previously  stated,  just  as  we  have  verified  the  SRL  with  the  acquisition  lifecycle,  we  must  also  validate 
the  optimization  models  against  real  systems  that  are  currently  under  development  within  the 
acquisition  environment.  Therefore,  Spiral  1  was  a  4-month  activity  that  applied  the  optimization 
models  to  real  systems  from  Northrop  Grumman  (Bethpage,  NY)  and  U.S.  Army  Armament  Research, 
Development  and  Engineering  Center  (Picatinny,  NJ). 
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3.2  Spiral  2  -  SERM  Development 

Step  2  in  Spiral  1  became  the  basis  for  Spiral  2  and  provided  valuable  insight  into  the  development  of  a 
scheduling  and  monitoring  tool  System  Earned  Readiness  Management  (SERM).  While  the  optimization 
models  are  unavoidably  mathematically  involved,  SERM  itself  is  envisioned  to  be  a  parsimonious 
management  tool.  It  will  measure  in  aggregate  terms  the  level  of  accomplishment  of  the  system 
development  process.  When  compared  to  the  development  plans  and  factoring  estimates  that  have 
been  prescribed  for  a  particular  system  under  development,  management  can  make  conclusions  on  its 
status  and  suggest  necessary  adjustments  to  correct  any  significant  deviations  that  will  impact 
acquisition  cost  and  schedule  (as  they  relate  to  developmental  maturity).  Thus,  Spiral  2  was  a  4-month 
effort  that  integrated  the  optimization  models  into  a  unified  evaluation  of  systems  development  to 
determine  planned,  budgeted,  and  actual  developmental  maturity;  and  determine  how  SERM  could  be 
used  to  better  understand  the  resource  implications  on  the  acquisition  lifecycle. 

3.3  Spiral  3/4  -  SERM  Verification  and  Validation 

The  success  of  implementing  these  models  depends  on  consistent  and  continuous  definition  of  needed 
capabilities  and  the  maturation  of  technologies  that  lead  to  disciplined  development  and  production  of 
systems  that  provide  increasing  capability  towards  a  material  concept.  A  fundamental  challenge  to 
defense  acquisition  is  that  the  ultimate  functionality  cannot  be  defined  at  the  beginning  of  the  program. 
Only  by  the  maturation  of  the  technologies,  matched  with  the  evolving  needs  of  the  user  can  they 
provide  the  user  with  capability.  These  final  two  spirals  were  an  8-month  effort  that  would  verify  and 
validate  SERM  with  their  associated  solution  techniques. 

3.4  Spiral  5  -  Methodologies  for  Multi-capability  Systems  (Future  Research) 

SERM  as  presently  configured  applies  directly  to  a  single  capability  system  undergoing  a  single-step 
development  process.  All  the  concepts  associated  with  it  must  be  re-examined  in  order  to  determine  if 
they  would  also  be  applicable  to  multi-capability  systems  undergoing  an  evolutionary  development 
process.  This  re-evaluation  is  necessary  when  we  consider  that  evolutionary  acquisition  is  the  adopted 
policy  and  that  most  systems  under  development  are  multi-functional.  This  portion  of  the  research 
started  in  2010  with  NPS  support  and  funding. 

The  approach  to  the  development  of  SERM  is  illustrated  by  the  diagram  in  Figure  2. 
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Figure  2:  Methodology  for  Developing  SERM 


Current  practices  and  gaps  were  determined  using  literature  review  and  discussions  with  practitioners 
through  meetings,  conferences  and  research  collaboration.  A  review  of  the  literature  on  the 
management  of  systems  development  was  used  to  identify  the  gaps  in  the  current  body  of  knowledge 
with  regard  to  scheduling,  monitoring  and  evaluation.  The  review  is  also  used  to  determine  if  methods 
used  in  similar  disciplines,  especially  project  management,  can  be  used  as  potential  foundations  for  any 
new  methodologies.  The  results  from  the  review  of  the  literature  were  validated  through  conversations 
with  people  who  are  actively  involved  in  the  development  of  high  novelty  systems  involving  high  to  very 
high  technological  components.  In  addition,  the  discussions  with  practitioners  was  used  to  determine  if 
any  cutting-edge  approaches  are  currently  being  considered  or  used  by  the  industry. 

Similar  information  was  also  sought  during  six  conferences  that  were  attended.  Finally,  while 
participating  in  research  collaborations  with  U.S.  Navy,  U.S.  Army,  Lockheed  Martin,  Northrop  Grumman 
Corporation  and  NASA  covering  efforts  related,  albeit  not  entirely,  to  this  research,  attempts  were  made 
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to  gather  information  on  how  systems  are  being  planned,  monitored  and  evaluated  with  a  view  to 
controlling  costs. 

The  assessment  of  current  practices  was  followed  by  a  selection  of  a  system  or  case  study,  which  can 
serve  as  a  platform  for  the  development  of  a  new  methodology  for  system  scheduling,  monitoring  and 
evaluation.  Such  a  system  must  be  novel,  involving  new  high  technology  components,  but  have  a 
manageable  number  of  critical  components  and  integrations  (7  plus  or  minus  2)  to  keep  the  discussions 
simple.  A  development  planning  model  SCODmin  (Magnaye  et  al.  2010)  was  applied  to  the  case  to 
identify  a  cost-minimizing  optimal  development  plan.  When  a  development  plan  has  been  determined, 
the  scheduling,  monitoring  and  evaluation  approach  was  formulated. 

4.  Development  of  Systems  Earned  Readiness  Management 
4.1  SERM  Background 

The  review  of  the  literature  on  the  management  of  systems  development  showed  that  many  tools  and 
methodologies  have  been  and  continue  to  be  developed  (Magnaye  2010).  The  planning  aspect  was 
addressed  in  the  seminal  work  by  Ramirez-Marquez  and  Sauser  (Ramirez-Marquez  and  Sauser  2009) 
through  the  maturity-based  model  SRLmax ■  When  it  comes  to  containing  costs,  at  best,  this  model  can  be 
used  indirectly  to  set  cost  targets  provided  the  temptation  to  develop  the  system  faster  than  it  needs  to 
be  is  avoided.  This  can  be  accomplished  by  limiting  the  availability  of  resources  allocated  for  each 
development  period.  This  is  easier  said  than  done.  Owners  of  systems  under  development  have  a 
tendency  to  see  their  products  completed  sooner  than  necessary.  For  example,  in  one  meeting  with 
personnel  from  the  U.S.  Navy,  it  was  observed  that  there  is  not  one  program  manager  who  would  not 
want  to  maximize  system  readiness.  This  is  probably  true  when  the  programs  fall  behind  schedule.  In 
order  to  avoid  a  tendency  to  spend  more  than  necessary  to  catch  up,  there  is  a  need  to  develop  a 
methodology  that  is  more  directly  targeted  at  formulating  a  system  development  plan  that  can  be  used 
to  minimize  the  cost  of  developing  a  system. 

To  address  this,  Magnaye,  et  al.  (2010)  suggested  a  similar  model  called  SCODmin.  This  model  was  used 
to  formulate  a  system  development  plan  which  can  identify  which  components  of  the  system  should  be 
matured  to  a  certain  level  during  a  given  time  period.  To  monitor  and  evaluate  the  progress  of  a  system 
under  development  in  a  logical  manner,  the  plan  must  be  translated  into  a  system  development 
schedule. 

The  need  for  a  scheduling,  monitoring  and  evaluation  method  is  supported  by  the  review  of  the 
literature  on  new  product  development.  It  observes  that  in  order  to  be  successful,  managerial  control 
(which  begins  with  setting  out  the  development  plan,  schedule  and  how  a  program  will  be  monitored 
and  evaluated)  must  be  applied  in  the  earlier  stages  of  new  product  development  (Bonner  et  al.  2002). 
The  ability  of  managerial  control  to  positively  influence  the  success  of  the  enterprise  is  also  reinforced 
when  it  is  applied  in  an  interactive,  flexible  and  responsive  manner  (lansiti  1995;  Bisbe  and  Otley  2004). 
Davila  (2000;  2009)  identified  a  couple  of  roles  that  managerial  control  plays:  1)  the  role  of  promoting 
goal  congruence  among  team  members  and  2)  the  less  traditional  role  of  reducing  uncertainty  (resulting 
from  novelty  and  high  technological  content)  by  enhancing  coordination  and  learning  through  the 
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creation  of  an  information  infrastructure.  This  is  most  important  during  the  design  and  planning  stages 
of  the  development  of  new  complex  products  or  systems.  This  need  for  control  through  coordination 
and  learning  using  timely  gathering  and  fast  dissemination  of  information  becomes  critical  to  success. 

Among  the  information  that  are  needed  are  changing  market  or  user  requirements,  technological 
improvements  (from  within  the  development  organization  and  from  competing  efforts  elsewhere)  and 
the  realities  of  systems  integration.  To  exercise  control,  knowledge  that  is  generated  during  systems 
development  has  to  be  reckoned  against  the  original  design  and  development  plans. 

Herstenstein  and  Platt  (2000)  observed  that  mechanisms  to  control  the  development  process  can  be 
classified  into  3  categories:  1)  the  position  of  product  development  in  the  organization,  2)  the  product 
development  process  and  3)  the  performance  measures  applied  during  the  process.  To  be  successful,  it 
has  been  observed  that  the  product  development  team  must  be  positioned  such  that  it  is  permanently 
headed  by  a  heavyweight  manager  who  reports  directly  to  a  senior  executive  (Cooper  2005)  (Clark  et  al. 
1987;  Clark  and  Fujimoto  1991)  Furthermore,  the  product  development  process  must  be  well  planned 
and  articulated  and,  along  with  the  performance  measures,  must  be  linked  to  the  strategy  of  the 
enterprise  (Cooper  and  Kleinschmidt  1987;  Cooper  and  Kleinschmidt  1993;  Brown  and  Eisenhardt  1995; 
Cooperand  Kleinschmidt  1995). 

Finally,  when  the  system  under  development  is  very  complex  and  has  high  levels  of  uncertainty  due  to 
their  novelty  and  high  technological  content  (Shenhar  and  Dvir  2007),  the  development  process  must  be 
in  the  optimizing  level  -  the  highest  level  of  process  maturity  as  defined  by  Karandikar,  Wood  and  Byrd 
(1992).  At  this  stage,  there  has  to  be  a  high  degree  of  control  over  the  process  and  the  major  focus  is  on 
the  application  of  process  metrics  and  lessons  learned  in  order  to  identify  quickly  the  problem  areas  and 
be  able  to  respond  promptly. 

The  literature  on  new  product  development  cited  above  suggests  that  to  have  success  during  the 
development  of  systems,  the  process  must  be  flexible  and  responsive  to  changes  in  technology  and 
requirements.  This  can  be  facilitated  through  an  interactive  managerial  control  system  that  promotes 
goal  congruence  and  enhance  learning  through  the  creation  of  an  information  infrastructure  with 
process  performance  metrics  that  are  linked  to  the  strategy  of  the  enterprise. 

Using  project  management  tools,  especially  Earned  Value  Management  (Barr  1996;  Brandon  1998),  as 
the  foundation  of  management  control  makes  sense  because  they  are  already  widely  accepted  in  this 
field.  Flowever,  they  are  inadequate  for  complex  systems  because  they  are  primarily  focused  on 
completing  tasks,  which  have  been  derived  from  engineering  development  plans  where  requirements 
and  designs  have  already  been  frozen.  At  best,  project  management  tools,  when  used  carefully,  can 
measure  very  precisely  the  accomplishments  and  evaluate  them  against  the  tasks  that  were  designated 
for  completion  at  a  particular  time.  Flowever,  they  cannot  be  used  to  manage  the  readiness  of  a  new 
technology,  let  alone  a  complex  system  composed  of  many  new  technologies  or  sub-systems  where  the 
concern  is  whether  or  not  the  new  technologies  or  system  are  maturing  as  required. 

With  regard  to  measuring  the  maturity  of  new  technologies  and  systems,  process  metrics  or 
performance  measures  for  systems  development  are  not  yet  fully  developed  (Suomala  2004).  Currently 
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available  tools  and  techniques  -  such  as  budgets,  Quality  Function  Deployment  (Hauser  and  Clausing 
1988),  Pugh's  Concept  Selection  Process  (Pugh  1991),  Kasser's  (Kasser  2004)  First  Requirements 
Elucidator  Demonstration  (FRED),  Integrated  Design  Model  (Vollerthun  2002),  Subsystem  Tradeoff 
Functional  Equation  (Shell  2003),  Design  for  Manufacturability  (Whitney  1988),  Design-Build-Test  Cycle 
(Clark  and  Fujimoto  1991)  and  Periodic  Prototyping  (Wheelwright  and  Clark  1992),  Cost  as  an 
Independent  Variable  or  CAIV  (Brady  2001)  and  Lean  Product  Development  Flow  (Oppenheim  2004)  -  for 
controlling  complex  product  development  are  fragmented  and  not  used  consistently  throughout  the 
process  (Pawar  and  Driva  1999).  More  than  an  unwillingness  to  bother  with  such  measures,  perhaps 
system  engineers  or  program  managers  do  not  use  them  consistently  because  they  find  them  to  be 
irrelevant  or  unable  to  address  the  needs  of  complex  high  uncertainty  products.  In  particular,  these 
tools  focus  only  on  measuring  specific  performance  aspects  of  the  system,  such  as  task  completion  or 
cost,  which  may  be  important  to  some  stakeholders  but  are  unable  to  show  if  the  system  is  maturing 
adequately  over  the  development  life  cycle.  Furthermore,  concentrating  on  the  measurement  of  these 
variables  can  lead  to  a  wrong  focus  in  terms  of  which  activities  are  prioritized.  Humans  tend  to  apply 
more  attention  to  activities  that  are  being  measured  and  rewarded  (Shuman  1995;  Chiesa  et  al.  2008; 
Blackburn  2009).  "What  gets  measured  is  what  gets  managed"  (Schmenner  and  Vollmann  1994). 
Therefore,  it  follows  that  when  traditional  project  management  tools  are  applied  to  assess  the 
completion  of  tasks,  a  program  manager  will  concentrate  on  meeting  costs  and  schedule  targets. 
Unfortunately,  in  the  case  of  high  novelty  and  high  technology  systems,  achieving  favorable  cost  and 
schedule  performances  alone  do  not  guarantee  that  the  system  is  maturing  as  planned.  This  is  because 
when  the  technology  development  tasks  were  identified,  there  was  no  certainty  yet  that  they  will  lead 
to  the  development  of  the  novel  technology.  They  were  scientific  educated  guesses.  What  may  happen 
is  that  by  focusing  on  the  project  tasks  and  ensuring  that  they  are  achieved  on  time  and  within  cost,  the 
program  manager  will  not  know  whether  or  not  the  required  readiness  has  been  achieved  until  much 
later.  That  is,  he  may  discover  that  some  of  the  tasks  have  to  be  repeated  using  a  different  approach  or 
materials.  Gaining  this  insight  on  a  timely  manner  is  crucial  to  containing  costs  because  applying  fixes 
later  is  always  more  costly.  Such  an  assessment  is  most  important  during  the  earlier  phases  of  the 
development  life  cycle  when  uncertainty  is  still  high  but  corrective  actions  are  still  manageable.  To 
encourage  a  system-wide  view  of  the  development  process,  the  program  manager  must  use  system- 
wide  maturity  measures. 

Scales  that  measure  readiness  levels  have  been  proposed  in  the  literature  (Sauser  et  al.  2006;  Sauser  et 
al.  2008;  Sauser  et  al.  2008;  Sauser  et  al.  2008;  Sauser  et  al.  2009,  July).  These  scales  -  TRL,  IRL  and  SRL 
-  have  gained  some  acceptance  in  the  field  of  systems  engineering  (Cuellar  2009;  Forbes  2009). 

The  review  of  the  literature  indicates  that  a  properly  constituted  managerial  control  that  is  focused  on 
the  maturity  of  technologies,  integration  elements  and  the  system  as  a  whole  is  important  to  the 
success  of  new  systems  development.  To  exercise  proper  control,  there  must  be  a  development  plan  to 
serve  as  the  foundation  for  a  scheduling,  monitoring  and  evaluation  method.  This  method  must  be 
interactive  and  encourage  learning. 
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4.2  Discussions  with  Industry  Representatives 

During  scheduled  meetings  with  members  of  the  industry  who  were  directly  involved  with  managing  the 
development  of  systems,  the  response  to  the  question  of  which  methodologies  they  use  for  planning, 
monitoring  and  evaluation  ranged  from  "Nothing"  to  "Earned  Value  Management  (EVM)".  However, 
the  latter  was  immediately  followed  by  the  observation  that  EVM  is  "inadequate  and  cumbersome". 
Upon  further  probing,  they  revealed  that  EVM  may  determine  if  the  tasks  are  being  accomplished  and  at 
what  costs  but  it  does  not  yield  any  concrete  information  as  to  whether  or  not  the  system  was  maturing 
at  an  acceptable  pace.  At  best,  they  can  only  hope  that  since  the  tasks  were  done,  the  system  has 
matured  accordingly.  Often,  this  was  not  the  case  because  the  technologies  and  the  system  itself  are  so 
new  and  technologically  advanced  that  no  one  really  knows  for  sure  how  much  maturity  has  been 
earned  for  the  system  under  development,  accomplished  tasks  notwithstanding. 

These  discussions  were  a  strong  indication  that  there  is  a  desire  to  have  a  planning,  scheduling, 
monitoring  and  evaluation  tool  specific  to  systems  development  which  highlights  system  readiness. 

4.3  Feedback  from  Conference  Presentations 

•  Technology  Maturity  Conference  2007  and  2008  -  this  has  been  the  primary  venue  for  the 
exchange  of  ideas  regarding  the  development  and  application  of  TRL.  It  is  usually  well  attended 
by  representatives  from  service  units  and  postgraduate  academic  institutions  of  the  Department 
of  Defense  (DoD),  the  Department  of  Energy  (DoE),  Department  of  Homeland  Security  (DHS), 
GAO  and  the  private  sector.  During  both  conferences,  some  of  the  observations  that  were 
gathered  were  the  on-going  attempts  to  refine  TRL,  use  it  to  manage  systems  development,  but 
also  the  clear  inability  of  TRL  to  measure  the  maturity  of  integration  links  and  systems. 

•  Acquisition  Research  Symposium  2008  and  2009  -  SRL  and  the  planning,  scheduling,  monitoring 
and  evaluation  methodology  for  systems  under  development  were  presented  during  these  DoD- 
sponsored  conferences.  The  remarks  received  during  and,  more  significantly,  after  the 
presentations  were  very  positive  and  reinforced  the  observations  that  were  gathered  from  the 
industry  representatives,  as  mentioned  earlier. 

4.4  Conceptual  Model 

With  the  results  of  the  research  in  mind,  the  conceptual  model  that  we  formulated  has  the  following 
steps  and  represented  in  Figure  3  by  the  shaded  areas: 

1.  Assign  costs  to  each  element  and  its  readiness  levels 

2.  Identify  the  optimal  development  plan 

3.  Translate  the  plan  into  a  system  development  schedule 

4.  Establish  a  readiness  management  baseline 

5.  Track  progress 

6.  Evaluate  performance 

7.  Apply  corrective  measures  (as  required) 

8.  Identify,  disseminate  and  apply  lessons  learned. 
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Figure  3:  Conceptual  Model 

The  cost  of  maturing  each  and  every  element  of  the  system  throughout  the  development  process  must 
be  estimated.  The  readiness  of  the  elements  is  determined  using  the  technology  readiness  level  (TRL) 
and  integration  readiness  level  (IRL)  scales.  The  data  on  cost  per  readiness  level  for  each  element  of  the 
system  is  the  main  input  to  the  SCODmin  model  (Magnaye  2010)  whose  solution  yields  an  optimal 
development  plan.  This  can  then  be  broken  down  into  the  individual  readiness  levels,  their  readiness- 
oriented  work  packages,  and  then  arranged  into  a  system  readiness  breakdown  structure  (SRBS). 


1.  System  Under  Development 

1.1  Critical  Technology  Element  1 

1.1.1  TRL  1 

1.1.2  TRL  2 

1.1.3  TRL  3 
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1.1. x  TRLx 

1.2  Critical  Integration  Element  1 

1.2.1  IRL  1 

1.2.2  IRL  2 

1.2.3  IRL  3 

1.2. x  IRL  x 


Figure  4:  Abbreviated  System  Readiness  Breakdown  Structure 

An  abbreviated  sample  is  shown  in  Figure  4.  Level  1  is  the  whole  system,  Level  2  indicates  its  critical 
elements  and  level  3  represents  their  readiness  levels,  which  have  to  be  accomplished.  Additional 
details  can  be  incorporated  to  show  the  work  packages  that  are  required  in  order  to  achieve  the 
readiness  levels. 

Using  the  SRBS  as  a  guide,  a  readiness  management  baseline  can  be  established.  This  show  the  tasks, 
when  they  should  be  achieved  to  reach  each  and  every  readiness  level  and  how  much  they  would  cost. 
A  work  package  is  awarded  an  earned  readiness  value  if  the  targeted  readiness  level  has  been  achieved, 
as  determined  by  an  independent  assessment  process,  which  the  organization  prescribes.  The 
performance  measurement  baseline  is  the  Budgeted  Cost  of  Readiness  Scheduled  (BCRS).  Actual 
Performance  is  represented  by  the  Budgeted  Cost  of  Readiness  Achieved  (BCRA)  and  its  cost  as  Actual 
Cost  of  Readiness  Achieved  (ACRA).  These  SERM  concepts,  their  relationship  to  each  other  and  how 
they  compare  to  similar  Earned  Value  Management  measures  are  illustrated  in  Figure  5. 
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Project  Management-Earned  Systems  Engineering  Management - 

allocated  Value  Management  Earned  Readiness  Management 


BCWP=budgeted  cost  of  work  performed  BCRA=budgeted  cost  of  readiness  achieved 

BCWS=budgeted  cost  of  work  scheduled  BCRS=budgeted  cost  of  readiness  scheduled 


Figure  5:  System  Earned  Readiness  Management  Concepts  Compared  to  Earned  Value  Management 

5.  Illustrative  Example 

To  show  how  the  conceptual  model  of  SERM  can  be  used,  we  applied  it  to  a  system  under  development. 
We  developed  a  case  around  the  sample  system  that  was  used  by  Magnaye  et  al(Magnaye  2010)  and 
added  additional  data  as  required.  This  case  was  based  on  a  space  robotic  system  that  was  developed 
by  NASA  but  was  later  aborted  in  favor  of  an  alternative  manned  approach.  The  paper  assumed  that 
this  aborted  system  has  been  revived  and  the  development  scenarios  were  laid  out  using  publicly 
available  information. 

5.1  Background 

The  test  case  that  was  written  -  Robotic  Servicing  Mission  for  the  Hubble  Space  Telescope  -  presented 
the  actual  development  of  the  robotic  servicing  system  that  was  estimated  to  cost  $1.3  billion.  The 
Hubble  Robotic  Servicing  and  De-orbit  System  (HRSDS)  was  worked  on  extensively  during  fiscal  year 
2004  and  fiscal  year  2005  at  a  cost  of  $700  million.  Over  1,000  persons  from  Goddard  Space  Flight 
Center  of  NASA,  Lockheed  Martin,  MDRobotics  and  several  other  contractors  completed  enough 
activities  such  that  the  program  went  through  a  very  successful  Preliminary  Design  Review  (PDR)  in 
March,  2005  (Whipple,  2009).  The  servicing  mission  was  estimated  to  have  duration  of  73  days  of  which 
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51  days  will  be  the  actual  servicing  of  the  observatory.  However,  later  that  year,  it  was  cancelled  after  it 
was  deemed  that  a  manned  servicing  mission  using  the  Space  Shuttle  was  safe  enough  and  had  a 
greater  chance  of  succeeding.  The  servicing  mission  (SM4)  was  completed  in  May,  2009. 

The  following  presents  a  backgrounder,  a  rationale  for  reviving  the  system  and  hypothetical  scenarios 
for  its  development. 

The  component  upgrades  and  replacements  to  the  Hubble  Space  Telescope  (HST)  that  Servicing 
Mission4  (SM4)  installed  in  May,  2009  have  degassed  and  the  observatory  successfully 
underwent  post-SM4  Servicing  Mission  Orbital  Verification.  HST  is  scheduled  for  de-orbit  and 
retirement  in  2014.  The  successor  to  the  HST-  the  James  Webb  Space  Telescope  (JWST)  -  is 
proceeding  and  it  is  expected  that  the  system  will  be  launched  in  2014.  However,  the  scientific 
community  expressed  some  doubts  as  to  whether  or  not  the  HST  should  be  retired  at  all.  One 
concern  stemmed  from  the  fact  that  the  JWST  can  only  capture  and  send  images  in  infrared 
wavelengths.  On  the  other  hand,  the  HST  operates  in  visible,  ultraviolet  and  near-infrared 
channels.  The  community  saw  advantages  in  having  all  modes  available.  The  other  concern  of 
the  scientific  community  was  that  without  HST,  any  delays  in  the  launch  of  the  JWST  will  lead  to 
a  period  when  a  gap  in  the  transmission  of  exploration  images  from  space  can  occur. 

A  series  of  consultations  within  NASA  and  between  the  agency  and  the  scientific 
community  led  to  the  conclusion  that  an  observatory  capable  of  handling  visual  and  ultraviolet 
images  such  as  the  HST  must  continue  operating  alongside  the  JWST.  Given  the  current 
economic  climate  at  the  time,  developing  and  launching  a  new  observatory  to  replace  the  HST 
was  not  approved.  Instead,  in  January,  2010,  Congress  authorized  NASA  to  send  another 
servicing  mission  to  maintain  or  even  upgrade  the  HST  by  2014. 

However,  it  was  noted  that  the  Space  Shuttle  Program  on  which  the  HST  servicing 
missions  have  relied  will  no  longer  be  in  operation  after  2012.  Sending  another  manned 
servicing  mission  before  then  was  also  not  feasible  because  the  manifests  for  the  remaining 
flights  were  already  full  and  the  new  space  shuttles  of  the  Constellation  Program  would  not  yet 
be  available  by  2014. 

To  solve  this  problem,  the  administrator  instructed  the  director  of  the  Goddard  Space 
Flight  Center  (GSFC)  to  put  together  a  plan  to  revive  and  complete  the  development  of  the 
Robotic  Servicing  Mission  (RSM)  which  was  originally  considered  for  the  SM4. 

In  late  August,  2009,  the  director  of  Goddard  Space  Flight  Center  (GSFC)  and  the 
manager  of  HST  Servicing  Mission  Operations,  Keith  Walyus,  submitted  an  updated  plan  to 
proceed  with  the  development  of  HRSDM  for  use  in  Servicing  Mission  5  scheduled  for  May,  2014. 
Based  on  consultations  and  collaboration  with  the  original  program  participants  or  their 
successor  entities,  the  plan  retained  the  original  technologies  and  architecture  of  the  system  (see 
Figure  6)  but  also  incorporated  the  latest  engineering  advances.  The  plan  estimated  that 
incremental  costs  to  mature  the  technologies  from  2010  to  2014  will  amount  to  $77.17  million 
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while  integrating  them  into  the  system  will  cost  another  $188.57  million  for  a  total  cost  of 
$265.74  million  (see  Table  1). 


Tech  1-  Remote  Manipulator  System  (RMS);  Tech  2  -  Special  Purpose  Dexterous  Manipulator  (SPDM); 
Tech  3  -  Electronic  Control  Unit  (ECU);  Tech  4  -  Autonomous  Grappling  (AG);  Tech  5  -  Autonomous 
Proximity  Operations  (APO);  and  Tech  6  -  Laser  Image  Detection  and  Radar  (LIDAR). 

Figure  6:  System  Concept  Diagram 


Table  I:  HRSDS  Incremental  (2010-2014)  Development  Costs 


Component 

Cost 

Comments 

Remote  Manipulator  System 

9.00 

Special  Purpose  Dexterous  Manipulator 

7.65 

Electronic  Control  Unit 

14.23 

Autonomous  Grappling 

21.50 

Autonomous  Proximity  Operations 

11.87 

Laser  Image  Detection  and  Radar 

12.92 

System  Integration 

188.57 

TOTAL 

265.74 

Satisfied  with  the  plans  and  cost  estimates,  the  new  NASA  director.  Dr.  Bolden,  informed 
the  White  House  who  promptly  advised  him  to  build  support  for  the  program  in  Congress.  After 
meeting  with  the  chairpersons  of  the  relevant  committees  in  both  chambers,  the  director 
concluded  that  congressional  support  for  HRSDM  will  be  greatly  enhanced  if  a  more  detailed 
multi-year  budget  tied  to  a  technology  maturation  plan  (as  called  for  by  the  Government 
Accountability  Office  -  GAO)  could  be  presented  during  the  supplementary  budgetary  hearings 
scheduled  for  late  September.  The  director  was  also  assured  that  by  developing  more  confidence 
in  the  ability  of  NASA  to  launch  HRSDM  successfully  in  2014,  a  waiver  may  be  granted  to  forego 
rebidding  of  the  program  elements.  In  effect,  the  prior  contractors  will  be  retained  and  thus  save 
considerable  time  in  restarting  the  program.  A  more  elaborate  technology  and  system 
development  plan  for  HRSDM  could  also  serve  as  a  model  for  future  programs  which  NASA  could 
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use  to  finally  address  the  concerns  of  GAO  as  it  continued  to  argue  for  the  need  for  technology 
system  metrics,  higher  maturity  of  technologies  that  go  into  NASA  systems  and  better  control  of 
development  costs. 

5.2  Application  of  SERM 

The  conceptual  model  for  the  Earned  Readiness  Management  approach  to  scheduling,  monitoring  and 
evaluation  of  the  development  process  was  applied  to  this  case.  The  first  step  was  to  breakdown  the 
cost  estimates  from  Table  I  to  determine  the  incremental  cost  of  maturing  every  element  through  each 
readiness  level.  These  are  shown  in  Tables  II  and  III. 


Table  II:  Incremental  Cost  to  Mature  Technologies  (cost  in  millions,  time  in  man-hours) 


Technology 

1 

2 

3 

4 

5 

6 

TRL  Level 

Effort 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

1 

2 

3 

4 

5 

6 

7 

$8.76 

127 

$4.67 

280 

$7.80 

450 

8 

$6.89 

476 

$4.21 

341 

$5.31 

236 

$1.23 

21 

9 

$9.00 

349 

$7.65 

432 

$7.34 

299 

$8.53 

568 

$1.89 

48 

$3.89 

300 

Table  III:  Incremental  Cost  to  Mature  Integrations  (cost  in  millions,  time  in  man-hours) 


Integration 

1,2 

1,3 

2,3 

2,4 

3,5 

4,5 

5,6 

IRL  Level 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

1 

2 

3 

$4.53 

200 

$1.23 

80 

4 

$5.81 

400 

$2.19 

380 

5 

$7.21 

658 

$5.95 

532 

6 

$1.00 

140 

$2.75 

164 

$9.00 

700 

$7.00 

621 

7 

$1.75 

180 

$2.00 

93 

$5.0 

25 

$5.40 

320 

$3.45 

324 

$12.00 

954 

$8.08 

862 

8 

$4.00 

300 

$4.00 

165 

$4.50 

320 

$6.32 

432 

$4.57 

400 

$14.32 

1021 

$10.03 

997 

9 

$6.00 

500 

$6.50 

389 

$5.50 

465 

$7.45 

690 

$6.78 

500 

$17.65 

1238 

$11.10 

1145 

Next,  we  apply  the  SCODmin  model  to  the  data  and  obtained  the  solution  or  development  plan  shown  in 
Table  IV. 
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Table  IV:  Optimal  Development  Plan 


Fiscal 

Target 

TRL 

IRL 

Year 

SRL 

1 

2 

3 

4 

5 

6 

1,2 

1,3 

2,3 

2,4 

3,5 

4,5 

5,6 

2014 

1.000 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

2013 

0.896 

9 

9 

9 

8 

9 

9 

9 

9 

9 

8 

8 

5 

7 

2012 

0.792 

8 

9 

9 

6 

9 

9 

9 

9 

9 

5 

8 

4 

6 

2011 

0.688 

8 

8 

9 

6 

9 

9 

8 

8 

7 

5 

7 

2 

4 

2010 

0.584 

8 

8 

8 

6 

7 

6 

7 

7 

7 

5 

6 

2 

4 

2010 

0.480 

8 

8 

7 

6 

6 

6 

5 

6 

6 

5 

6 

2 

2 

From  the  optimal  development  plan,  we  prepared  the  system  development  schedule  by  showing  the 
system  readiness  breakdown  structure  along  the  vertical,  the  timetable  on  the  horizontal  and  the  costs 
in  the  cells  formed  by  their  intersections.  The  schedule  from  2004  to  2014  is  shown  as  Table  V. 
According  to  this,  work  on  the  Electronic  Control  Unit  (element  3),  Autonomous  Proximity  Operations 
module  (element  5),  the  integration  links  between  elements  1&2,  1&3,  2&3  and  5&6  are  to  resume  in 
Fiscal  Year  2010  requiring  a  budgetary  allocation  of  $20,230  million.  The  elements  and  integration  links 
that  should  be  worked  on  for  the  succeeding  years  can  also  be  identified  from  Table  V  along  with  their 
costs.  Altogether,  including  the  $700  million  that  was  already  spent  for  development  from  2004  to  2005 
followed  by  mothballing  until  2009,  the  system  under  development  will  cost  $965.74  million. 

There  are  situations  where  the  optimization  algorithm  suggested  that  work  on  an  element  of  the  system 
be  put  on-hold  during  some  years  and  resumed  at  a  later  year.  For  example,  the  integration  link 
between  elements  2&3  is  advanced  to  an  IRL  of  7  by  2010  but  no  work  on  it  is  called  for  during  2011.  It 
is  resumed  only  in  2012.  Similar  situations  exist  for  the  integration  links  between  elements  3&5  and 
5&6.  This  could  mean  that  workers  and  equipment  may  be  idle  for  a  year.  It  is  not  a  big  issue  if  they  can 
be  diverted  to  other  tasks.  Otherwise,  it  represents  opportunity  costs  (due  to  foregone  revenues)  and 
administrative  ones  associated  with  laying-off  workers  and  re-hiring  them  later.  This  should  be  avoided 
unless  there  are  genuine  technical  justifications  for  it. 
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To  avoid  this  problem,  the  development  team  may  be  allowed  to  start  such  affected  activities  early  to 
maintain  continuity.  For  example,  instead  of  waiting  until  2012  as  prescribed  by  the  algorithm,  work  on 
reaching  an  IRL  of  8  for  the  integration  link  between  elements  2&3  can  be  moved  forward  to  2011.  The 
program  managers  and  the  owners  of  the  system  must  weigh  the  trade-off  between  incurring  some 
expenditures  earlier  than  originally  planned  versus  the  administrative  and  technical  costs  of  putting  the 
affected  elements  of  the  system  "on-hold"  until  it  can  be  re-started  at  the  later  prescribed  date.  The 
revised  system  development  schedule  is  shown  in  Table  VI. 

The  actual  progress  of  the  development  process  can  be  measured  on  a  regular  basis,  compared  to  the 
original  schedule  and  the  readiness  and  cost  performance  measures  using  SERM  can  be  calculated  to 
identify  problem  areas  and  formulate  remedial  measures. 
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5.3  Managerial  Implications  and  Future  Research 

This  report  presented  an  approach  which  can  operationalize  the  implementation  of  a  system 
development  plan  such  as  the  one  obtained  from  the  cost  minimization  optimization  model  ( SCODmin ) 
proposed  by  Magnaye  et  al  (2010).  The  approach  -  System  Earned  Readiness  Management  -  provides 
program  or  system  engineering  managers  with  the  tools  to  schedule,  monitor  and  evaluate  the 
completion  of  tasks  aimed  at  achieving  the  planned  maturity  of  the  system  as  measured  by  SRL.  This  is 
the  primary  contribution  of  this  paper.  With  SERM,  it  is  now  possible  to  exercise  a  more  effective 
maturity-focused  managerial  control  over  the  process  of  developing  new  systems  as  has  been  suggested 
by  the  Government  Accountability  Office  and  practitioners  themselves.  SERM  also  reinforces  the  ability 
of  program  managers  to  define  and  analyze  the  development  of  individual  technologies  that  go  into  a 
system  not  as  isolated  projects  but  as  critical  parts  of  an  integrated  unit  (Forbes  et  al.  2009). 

SERM  can  be  used  in  an  interactive  manner  that  can  also  be  integrated  with  the  learning  processes  of 
the  development  organization.  This  is  important  when  the  system  involves  a  high  level  of  novelty  and 
technological  contents.  Such  a  system  will  undergo  multiple  designs,  technology  choices  and  cost 
estimates,  generating  valuable  insights  and  lessons.  These  can  be  captured  by  an  iterative  application  of 
SERM  as  the  novel  technologies,  architecture  and  functionalities  within  the  system  become  clarified  and 
better  understood.  For  example,  new  more  accurate  cost  data  can  be  entered  into  the  SCODmin 
optimization  model  to  generate  a  revised  development  plan  which  can  then  be  translated  into  the 
system  breakdown  structure,  schedule  and  so  on.  The  same  could  be  done  with  changes  in  technology 
choices,  system  architecture  or  capabilities. 

In  accordance  with  the  wishes  of  the  practitioners  that  we  consulted  throughout  the  research  process, 
SERM  is  both  simple  and  familiar.  The  system  development  schedule  is  in  the  form  of  a  GANTT  chart 
which  is  used  routinely  in  project  management.  The  system  breakdown  structure  (Figure  4)  is  very 
similar  to  a  project  work  breakdown  structure  (WBS)  while  determining  readiness  and  cost  variances 
using  SERM  is  almost  the  same  as  in  project  earned  value  management  (see  Figure  5). 

The  effectiveness  of  SERM  in  facilitating  better  managerial  control  over  the  development  of  a  novel  high 
technology  system  will  be  greatly  enhanced  by  a  thorough  verification  and  validation  of  the  metrics 
which  serve  as  its  foundation.  These  are  the  TRL,  IRL  and  SRL  scales.  They  must  be  applied  to  a  wide 
cross-section  of  technologies  and  systems  across  all  the  relevant  domains  which  include,  but  are  not 
limited  to,  strategic  national  defense,  aerospace,  software,  energy,  transport,  environment  and 
economic  systems.  The  primary  goal  would  be  to  determine  which  values  of  SRL  correspond  to  which 
phases  of  the  development  life  cycle  for  each  domain. 

SERM  itself  must  be  validated  by  examining  its  practicality  when  managing  the  development  of  a 
system.  This  would  involve  a  longitudinal  research  study  of  a  diverse  collection  of  systems  from 
beginning  to  deployment  and  disposal.  Earned  Value  Management  for  projects  did  not  become  a 
mature  concept  until  it  was  experimented  with  by  the  students  and  faculty  of  the  US  Air  Force  Institute 
of  Technology  (Abba  1997;  Abba  2001).  Perhaps  SERM  for  systems  should  also  be  subjected  to  the  same 
amount  of  scrutiny  by  the  academic  institutions  and  contractors  associated  with  the  Departments  of 
Defense  and  Energy  as  well  as  NASA. 
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Our  research  showed  that  there  is  a  desire  among  program  and  systems  engineering  managers  to  have  a 
methodology  for  exercising  managerial  control  over  a  novel,  high  technology  system  undergoing 
development.  Such  a  methodology  must  be  based  on  a  system-wide  view,  focused  on  readiness  or 
maturity  of  the  system,  which  can  be  employed  in  an  interactive  and  informative  manner.  Above  all 
else,  it  must  be  simple  and  have  a  familiar  feel  or  the  practitioners  will  be  reluctant  to  employ  it. 

Such  a  desire  is  justified  by  the  conclusions  that  one  can  draw  from  the  literature  on  managerial  control 
of  new  product  development.  Our  review  showed  that  such  a  control  mechanism  is  important  to 
success  so  long  as  it  is  applied  early  in  the  process  or  life  cycle,  applied  in  an  interactive  manner  and  can 
enhance  coordination  and  learning  through  the  creation  of  an  information  infrastructure. 

With  these  in  mind,  we  crafted  a  methodology  called  System  Earned  Readiness  Management  that  is 
patterned  after  tools  that  were  generally  accepted  in  the  project  management  domain.  What 
distinguishes  SERM  is  that  the  information  presented,  analyzed  and  used  for  decision-making  revolves 
around  metrics  on  readiness  or  maturity  of  a  novel  high  technology  system,  its  critical  technology 
elements  and  the  integrations  among  them.  We  illustrated  the  use  of  SERM  by  applying  it  to  a  sample 
space  system  -  the  Hubble  Robotic  Servicing  and  De-orbit  System  -  with  actual  but  disguised  data  from 
2004  to  2009  and  hypothetical  scenarios  from  2010  to  2014. 

SERM  can  be  greatly  refined  and  validated  if  it  can  be  applied  to  actual  systems  that  are  about  to 
undergo  development.  This  calls  for  longitudinal  studies,  which  we  hope  to  initiate  with  the 
cooperation  of  a  defense  contractor  and  an  oil  services  firm. 

6.  Project  Accomplishments 

6.1  Publications 

6.1.1  Journal 

Tan,  W.,  J.  Ramirez-Marquez,  and  B.  Sauser.  (2010).  A  Probabilistic  Approach  to  System  Maturity 
Assessment.  Systems  Engineering  (accepted) 

Sauser,  B.,  R.  Gove,  E.  Forbes,  and  J.  Ramirez-Marquez.  (2010).  Technology  Integration  Maturity 
Metrics:  Development  of  an  Integration  Readiness  Level.  Information,  Knowledge,  Systems 
Management  (accepted) 

6. 1.2  Conference  Proceedings 

Tan,  W.,  B.  Sauser,  and  J.  Ramirez-Marquez.  (2009).  Monte-Carlo  Simulation  Approach  for  System 
Readiness  Level  Estimation.  International  Symposium  of  the  International  Council  on  Systems 
Engineering.  July  20-23,  Singapore  (Brian  Mar  Best  Student  Paper) 

Sauser,  B.,  E.  Forbes,  M.  Long,  and  S.  McGrory.  (2009).  Defining  an  Integration  Readiness  Level  for 
Defense  Acquisition.  International  Symposium  of  the  International  Council  on  Systems  Engineering. 
July  20-23,  Singapore  (Best  Paper  in  Government  Domain) 
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Magnaye,  R.,  B.  Sauser,  J.  Ramirez-Marquez,  and  W.  Tan.  (2009).  Using  a  System  Maturity  Index  to 
Monitor  and  Evaluate  the  Development  of  Systems.  Acquisition  Research  Symposium.  May  13-14, 
Monterey,  CA 

Cuellar,  R.,  and  B.  Sauser.  (2009).  Dynamic  Multipoint  Optimization  Application  to  Corporate  Portfolio 
Management.  Acquisition  Research  Symposium.  May  13-14,  Monterey,  CA 

6.2  Presentations 

"Systems  &  Change:  Understanding  System  Maturity"  General  Dynamics  Corporate  Leadership  Forum, 
Webinar  (seminar),  February  23,  2010  (invited) 

"System  and  Integration  Readiness  Levels  for  Defense  Acquisition."  INCOSE  Heartland  Chapter, 
Webinar  (seminar),  November  3,  2009  (invited) 

"Dynamic  Modeling  of  Programmatic  and  Systematic  Interdependence  for  System  of  Systems 
Acquisition."  National  Defense  and  Industry  Association  Systems  Engineering  Conference,  San 
Diego,  CA,  October  29,  2009 

"Linking  Systems  Engineering  Artifacts  with  Complex  System  Maturity  Assessments."  National  Defense 
and  Industry  Association  Systems  Engineering  Conference,  San  Diego,  CA,  October  28,  2009 

"System  Maturity  Assessment  for  Decision  Support  in  Lifecycle  Acquisition."  INCOSE  Chesapeake 
Chapter,  Applied  Physics  Laboratory,  Johns  Hopkins  University,  October  3,  2009  (invited) 

"Systems  Maturity  Assessment  for  Defense."  National  Security  Agency  Learning  Seminar,  September 
9,  2009  (invited) 

"System  (of  Systems)  Acquisition  Maturity  Models  and  Management  Tools."  Office  of  the  Secretary  of 
Defense  Software  Collaborators  Webinar  (seminar),  August  18,  2009  (invited) 

"Defining  an  Integration  Readiness  Level  for  Defense  Acquisition."  International  Symposium  of  the 
International  Council  on  Systems  Engineering,  Singapore,  July  21,  2009 

"System  Maturity  Assessment  for  Decision  Support  in  Lifecycle  Acquisition."  University  of  Alabama  in 
Huntsville  College  of  Business  Administration,  Huntsville,  AL,  July  7,  2009  (invited) 

"A  Review  of  Frameworks  and  Models  from  Maturity  to  Collaboration  in  Systems  and  System  of 
Systems  Engineering."  Texas  A&M  University  Department  of  Industrial  and  Systems  Engineering 
Seminar,  College  Station,  TX,  June  29,  2009  (invited) 

6.3  Awards 

Best  Paper  in  Government  Domain 

International  Symposium  of  the  International  Council  on  Systems  Engineering,  Singapore,  July  2009 

Brian  Mar  Best  Student  Paper 

International  Symposium  of  the  International  Council  on  Systems  Engineering,  Singapore,  July  2009 
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Note:  From  the  potential  four  best  papers  given  at  the  International  Symposium  of  the  International 
Council  on  Systems  Engineering,  we  won  two  out  of  the  four. 

6.4  Knowledge  Transfer  to  Industry /Government 

6.4. 1  U.S.  Army  Armament  Research  Development  and  Engineering  Center  (ARDEC) 

We  have  built  strong  working  relationship  with  the  U.S.  Army  ARDEC  System  Engineering  Director.  This 
relationship  has  resulted  in  an  effort  to  develop  a  guide  and  tool  for  determining  a  systems  maturity 
readiness  and  potential  for  making  efficient  and  effective  life-cycle  acquisition  and  operational 
decisions.  In  addition,  an  ARDEC  employee  will  be  utilizing  our  research  outputs  in  a  Six  Sigma  Black  Belt 
Project  in  order  to  benchmark  their  utilization  in  program  review  processes. 

6.4.2  Northrop  Grumman  /  USN  PMS  420 

Northrop  Grumman  has  worked  in  partnership  with  the  US  Navy's  Littoral  Combat  Ship  Mission  Modules 
Program  Office  (PMS  420),  to  implement  the  SRL  methodology  across  the  Mission  Modules 
development  effort.  Since  its  roll-out  to  the  program  in  September  of  2007,  SRL's  role  has  steadily 
evolved  to  become  a  vital  component  of  both  system  technical  development  status  monitoring  and  on¬ 
going  resource  allocation  decisions.  From  inception,  the  Mission  Modules  Program  has  been  chartered 
with  leveraging  a  large  number  of  existing  DOD  programs  of  record  and  COTS/GOTS  equipment  and 
integrating  them  together  to  provide  enhanced  capabilities  and  data  sharing.  Due  to  the  inherently 
mature  incoming  system  components,  the  need  for  analysis  and  monitoring  of  overall  development 
maturity  beyond  the  TRLs  was  acute.  By  quantitatively  evaluating  the  maturity  of  the  complex  network 
of  integrations  in  concert  with  the  components  they  connect,  the  IRL  scale  and  SRL  methodology  have 
proved  to  be  invaluable.  The  scale  provides  a  common  dashboard  view  of  true  system  maturity  enabling 
decision  makers  better  understand  current  status  and  mitigate  emerging  risks  in  the  systems  integration 
activity.  The  concept  is  also  being  expanded  for  use  in  analyzing  future  technology  insertion  options  and 
program  development  costs.  This  work  with  Northrop  Grumman  and  the  U.S.  Navy  has  resulted  in  the 
SD&ML  begin  directly  funded  by  the  U.S.  Navy  PMS  420  through  the  Systems  Engineering  Research 
Center. 

6.5  Other  Related  Activities: 

6.5. 1  Systems  Maturity  Assessment  Roundtable 

On  March  12,  2009,  a  Roundtable  was  held  in  Washington,  DC  with  the  purpose  of  providing  system 
designers  and  developers,  program  and  project  managers,  and  researchers  a  platform  to  discuss  and 
disseminate  emerging  knowledge  in  systems  maturity  indices  (beyond  TRL).  The  objective  was  to  create 
a  community  of  practitioners  and  researchers  focused  on  new  knowledge  in  system  maturity  indices  and 
assessment. 

For  the  first  half  of  the  day,  presentations  were  made  from  stakeholders  on  the  emerging  challenges 
and  potential  solutions  in  systems  maturity  indices  and  assessment.  For  the  second  half  of  the  day, 
breakout  groups  were  asked  to  address  these  four  questions  with  respects  to  the  future  of  systems 
maturity  assessment: 
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1.  What  are  the  real  questions? 

2.  What  do  we  know? 

3.  What  do  we  need  to  know? 

4.  What  could  we  do  to  learn  that? 


Copies  of  the  presentations  and  a  summary  report  of  the  outcomes  can  be  found  at 
http://www.SystemReadinessLevel.com,  under  SMA  Roundtable. 

6.5.2  Web  Page 

From  the  birth  of  this  research,  we  have  believed  in  an  open  academic  model  of  sharing  our  research 
outcomes  in  the  broadest  sense  possible.  Thus,  we  have  created  a  web  site, 

http://www.SystemReadinessLevel.com,  for  the  distribution  of  our  research  results.  At  this  web  site  you 
will  find:  Research  Overview;  Publications/Presentations/News,  Research  Projects;  and  Contact 
Information. 

6.5.3  Conferences  Attended 

•  Systems  Engineering  Conference,  National  Defense  and  Industry  Association,  San  Diego,  CA, 

October  2009 

•  Acquisition  Research  Symposium,  Monterey,  CA,  May  2009. 

6.5.4  Student  Research  Supported/Supervised 

Our  funding  from  the  Navel  Postgraduate  School  as  afforded  us  the  privilege  to  support  one  Ph.D. 
student  to  assist  in  the  execution  of  this  research.  But,  it  has  also  allowed  us  the  ability  to  attract 
graduate  student  to  pursue  related  and  supportive  research.  These  students  are: 

•  Ana  Lisbeth  Cone  ho.  M.S.  Student.  "Functionally  Equivalent  COTS  for  Optimal  Component 

Substitution  within  System  Evolution  Planning" 

•  Romulo  Magnaye.  Ph.D.  Student.  Robert  Crooks  Stanley  Fellow.  "Using  a  System  Maturity  Scale 

to  Monitor  and  Evaluate  the  Development  of  Complex  Systems" 

•  Weiping  Tan.  Ph.D.  Student.  NPS  Supported.  "A  Probabilistic  Approach  to  System  Maturity 

Assessment." 

6.5.5  Student  Projects  Supervised 

Within  the  School  of  Systems  and  Enterprises  at  Stevens  Institute  of  Technology,  students  are 
encouraged  to  complete  a  3-credit  special  problems  project  as  part  of  their  course  requirements. 
Because  of  the  success  of  this  research,  we  have  been  able  to  attract  a  number  of  students  to  pursue 
projects  related  to  SRL  and  related  topics.  Flere  is  a  list  of  those  students  and  projects: 

Sweeton,  J.  (2009).  "Transitioning  Innovations  into  an  Agile  System  Analysis  of  Cost  and  Improving 
Communication."  M.S.  Special  Problems  in  Systems  Engineering. 
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Jumbo,  L.  (2009).  "Evaluation  of  Selected  DOD  Systems  Development  Using  the  System  Readiness  Level 
(SRL)  Concept"  M.S.  Special  Problems  in  Systems  Engineering. 

Snow,  G.  (2009).  "The  Use  of  System  Maturity  Indices  to  Assess  &  Manage  Risk  in  an  Open  System  from 
Development  through  Production."  M.S.  Special  Problems  in  Systems  Engineering. 

Lin,  D.  (2009).  "Develop  a  Producibility  Readiness  Level  to  Complement  System  Readiness  Level  within 
Defense  Acquisition  Systems."  M.S.  Special  Problems  in  Systems  Engineering. 

Van  Nostrand,  A.  (2009).  "What  can  Constellation  Learn  from  Taking  a  Soft  Systems  View  of  the 
Reliability  Success  of  Apollo?"  M.S.  Special  Problems  in  Systems  Engineering. 
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